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(57) Abstract: The present invention relates to the 
field of hand-held devices that are equipped wit a 
processor and a digital camera for capturing motion 
video or still images. Images captured by the cam- 
era are used to determine motion of the hand-held 
device. A resulting motion signal is used as input to 
a user interface. Displayed images can be scrolled, 
zoomed or rotated by moving the hand-held device. 
The motion signal can also be used as input for a 
graphical user interface to move a cursor or other 
object of the graphical user interface over the dis- 
play. The invention relates further to a hand-held 
device provided with means for sensing motion, a 
display, a keypad with at least a first- and a second 
key, and a graphical user interface with objects and a 
cursor. The hand-held device also comprises means 
for transforming the sensed motion of the handheld 
device into a signal suitable for moving the cursor 
over the display. 



BEST AVAILABLE COPY 



wo 2^/066615 Al lillllilllillllllllllllllfti 



Eurasian patent (AM, AZ, BY, KG, KZ, MD, RU, TJ, TM), For two-letter codes and other abbreviations, refer to the "Guid- 

European patent (AT, BE, BG, CH, CY, CZ, DE, DK, EE, once Notes on Codes and Abbreviations'* appearing at the begin- 

ES, H, FR. GB, GR, HU, IE, IT, LU, MC, NL, PT, SE, SI, ning of each regular issue of the FCT Gazette, 
SK, TR), OAPI patent (BF, BJ. CF, CG, CI, CM, GA, GN, 
GQ, GW, ML. MR, NE, SN, TD. TG). 

Published: 

— with international search report 



wo 2004/066615 



:T/£P2003/000607 



IMAGE CONTROL 

The present invention relates to the field of hand-held 
5 devices that are equipped with a processor and a digital 
camera for capturing motion video or still images, in 
particular such devices that further comprise a display 
for displaying images or a graphical or character based 
user interface. 

10 

BACKGROUND ART 

Hand-held devices provided with digital image capturing 
equipment, digital processing power and high resolution 
15 displays are becoming increasingly more common in a wide 
variety of uses. 

For example, small mobile phones are recently being 
equipped with digital cameras and relatively small high 
resolution LCD screens . Hand-held computers commonly 
called "^"personal digital assistants" (PDA) are also 
available and are typically equipped with small high 
resolution display screens and have slots for receiving 
e.g. a digital camera. Similarly, commxinicators having 
both cellular communication and computer capal>ilities are 
available, typically having small display screens and an 
inbuilt or detachable digital camera. These small, hand- 
held devices do not, and cannot, conveniently have 
conventional input devices, such as a computer mouse and 
other control keys. Therefore, conventional personal 
computer interfaces, which also have their own drawbacks, 
are not suited for these small hand-held devices. 

As a result, there are significant limitations on using 
35 such small hand- held devices in both obtaining output, 
e.g. viewing data on the display screen, and in inputting 
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commands, e.g. chajiging the area viewed on the display 
screen or controlling the performance of a particular 
parameter associated with the device. Further, given the 
limited area available, not only on the display screen but 
5 also on the entire device, adding additional control keys, 
etc., is both difficult and burdensome to a user requiring 
two hand operation of the device. 

US 6466198 discloses a system and method for view 
10 navigation and magnification of the display of hand-held 
devices in response to the orientation changes along only 
two axes of rotation as measured by sensors inside the 
devices. The view navigation system can be engaged and 
controlled by simultaneously pressing switches on both 
15 sides of the hand-held device. Miniature sensors like 
accelerometers, tilt sensors, or magneto- resistive 
direction sensors sense the orientation changes. These 
miniature sensors are presently not typically standard 
equipment for hand-held devices. Thus, such sensors add 
20 cost, use precious space and add weight. 

The present invention is directed toward overcoming one or 
more of the above -identified problems. 

25 DISCLOSURE OF THE INVENTION 

On this background, it is an object of the present 
invention to provide a hand-held device of the kind 
referred to initially, which allows user input with the 
30 same hand that holds the device, without requiring the 
dedicated sensatory equipment used by prior art hand-held 
devices . 

This object is achieved in accordance with claim 1, by 
35 providing a hand-held device comprising a processor, a 
digital camera for capturing motion video or still images. 



wo 2004/066615 




:T/EP2003/000607 



and means for transforming a signal from the camera into a 
motion signal indicative of the motion of the hand-held 
device . 

5 Thus, by using a sensor that is available to start with in 
many hand-held devices namely a digital camera for a 
secondary use, namely creating a motion signal indicative 
of the motion of the hand-held device, a hajad-held device 
with motion sensing is provided in a economical and 

10 reliable manner. 

The hand-held device may further comprise a user interface 
in which motion of the hand-held device is - through the 
motion signal derived thereof - used as a user input. 



The hand-held may further comprise a display, preferably a 
display suitable for displaying captured images. 

Motion of a given type of the hand-held device can be used 
20 to manipulate images shown at least in part on the " 
display, preferably by moving the images in a manner 
substantially corresponding to the movement of the hand- 
held device. 

25 Different types of motion the hand-held device can e.g. be 
used to move, and/ or zoom, and/or expand/ col lapse and/ or 
rotate images displayed on the display. 

Motion substantially parallel to the plane of the display 
30 of the hand-held device cam be used to scroll am image 
displayed on the display. Motion substantially 
perpendicular to the plane of the display can be used to 
zoom an image displayed on the display. Rotational motion 
of the hand-held device can be used to rotate am image 
35 displayed on the display. 



15 
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The images that are manipulated can e.g. be images that 
were previously captiired by the camera. 

The movement of image can be inverted with respect to 
5 motion of the hand-held device, since some users may 
prefer this- 

The user interface may comprise a graphical user interface 
and motion of the hand-held device can be used as an input 
10 to the graphical user interface. 

Motion of the hand-held device can be used to manipulate 
an object displayed by the graphical user interface, 
preferably by moving the object in a manner substantially 
15 corresponding to the motion or to the inverted motion of 
the hand-held device, whereby the object displayed by the 
graphical user interface can be an icon, a dialogue box, a 
window, a menu or a pointer. 

20 Motion of a given type of the hand-held device can be used 
to move, and/or zoom, and/ or expand/collapse and/or rotate 
objects displayed by the graphical user interface. 

Motion substantially parallel to the plane of the display 
25 of the hand-held device can be used to scroll an object 
displayed by the graphical user interface. Motion 
substantially perpendicular to the plane of the display 
can be used to zoom an object displayed by the graphical 
user interface. Rotational motion of the hand-held device 
30 can be used to rotate an object displayed by the graphical 
user interface. 

The digital camera can be an inbuilt camera or can be a 
detachable camera. The camera may be movable relative to 
35 the hand-held device. 
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The means for transforming a signal from the camera into a 
motion signal preferably derives the motion signal from 
changes between succeeding images captured by the camera. 

5 The camera can be equipped with an autofocus system, 
whereby the focusing setting of the autofocus system can 
be used for detecting movement in the camera direction. 

The hand-held device may further comprise at least one 
10 key. The functionality of a motion type can be dependent 
on the state of the at least one key. 

Rotational motion of the hand-held device about an axis 
substantially perpendicular to the display may be used to 
15 cause an inverse rotational movement of the image or 
graphical user interface object relative to the display, 
preferably in a manner such that the image or object is 
static with respect to the fixed coordinate system in 
which the hand-held device is situated. 

20 

The motion signal can be used to adjust device settings, 
such as sound settings, keypad settings eind display 
settings . 

25 The hand-held device may further comprise a keypad with at 
least a first and a second key and the graphical user 
interface comprising a cursor, whereby motion of the hand- 
held device csin be used to position the ciirsor over an 
object of the graphical user interface and primary 

30 functions associated with the object concerned can be 
activated by pressing the first key and secondary 
functions associated with the object of the concerned can 
be activated by pressing the second key. 

35 The fionctionality of the first key can be associated with 
selection and activation of objects of the graphical user 
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interface, and the functionality of the second key can be 
associated with calling up a context-sensitive menu. 

The selection of the object concerned can be perforined by 
5 pressing and releasing the first key. Activation of the 
object concerned can be performed by pressing and 
releasing the first key twice in rapid succession. Moving 
or resizing of the object concerned can be performed by 
holding down the first key while moving the hand-held 
10 device to move the cursor. 

The first key and the second key cam be softkeys, whereby 
the current functionality of the softkeys is shown in the 
display, preferably in dedicated fields of the display. 

15 

The first key can be placed below the display on the left 
side of the latter, preferably proximate to lower edge of 
the display, and the second key can be placed below the 
display on the right side of the latter, preferably 
20 proximate to lower edge of the display. 

It is another object of the present invention to provide 
an improved method for proving user input to hand-held 
devices. This object is achieved by providing a method for 
25 creating user input for a hand-held device that has a 
processor, a user interface and a digital camera for 
capturing motion video or still images comprising the 
steps of: 

determining motion of the hand-held device from 
30 the camera signal; 

using the determined motion of the hand-held device 
as an input for the user interface. 

It is yet another object of the present invention to 
35 provide a use of a digital camera of a hand-held device 
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that has a processor to produce a motion signal indicative 
of motion of the hand-held device. 

A further object of the invention is to provide a hand- 
5 held device with an improved graphical user interface. 
This object is achieved by providing a hand-held device 
comprising a processor, means for sensing motion of the 
hand-held device, a display, a keypad with at least a 
first- and a second key, a graphical user interface with 
10 objects and a cxirsor, and means for transforming the 
sensed motion of the handheld device into a signal 
suitable for moving the cursor over the display. 

By controlling the position of a cursor through motion of 
15 the hcoidheld device it becomes possible to provide a user- 
friendly cursor controlled graphical user interface for 
hand-held devices. 

Preferably, motion of the hand-held device is used to 
20 position the cursor over objects of the graphical user 
interface and primary functions associated with the object 
concerned are activated by pressing the first key and 
secondary functions associated with the object concerned 
are activated by pressing the second key. 

25 

The functionality of the first key can be associated with 
selection and activation of objects of the graphical user 
interface, and the functionality of the second key can be 
preferably associated with calling up a context-sensitive 
30 menu . 

Selection of the object concerned is preferably performed 
by pressing and releasing the first key, and activation of 
the object concerned is preferably performed by pressing 
35 and releasing the first key twice in rapid succession. 
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Moving or resizing of the object concerned can be 
performed by holding down the first key while moving the 
hand-held device to move the cursor and the object 
concerned in \mison therewith. 

5 

The first key and the second key can be softkeys, whereby 
the ciirrent fiinctionality of the softkeys is shown in the 
display, preferably in dedicated fields of the display. 

10 The first key can be placed below the display on the left 
side of the latter, preferably proximate to lower edge of 
the display, and the second key can be placed below the 
display on the right side of the latter, preferably 
proximate to lower edge of the display. 

15 

The means for transf canning motion of the handheld device 
into a signal suitable for moving the cursor over the 
display may further comprise a tilt sensor and/or an image 
capturing device and/or an accelerometer . 

20 

The hand-held device according may further comprise means 
to transfo2rm a signal from the image capturing device, 
i.e. camera and/or tilt sensor and/ or accelerometer into a 
position signal for the cursor. 

25 

The means for transforming a signal from the camera into a 
motion signal preferably derives- the motion signal from 
changes between succeeding images, or parts of succeeding 
images captured by the camera. 

30 

The camera may have an autofocus system, whereby the 
focusing setting of the autofocus system is used for 
detecting movement in the camera direction. 
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The graphical user interface may include one or more of 
the following object types: icons, dialogue boxes, 
windows , menus , pointers . 

5 Further objects, features, advantages and properties of 
the hand- held device, method for proving user input and 
use of a digital camera in a hand-held device according to 
the invention will become apparent from the detailed 
description . 

10 

BRIEF DESCRIPTION OF THE DRAWINGS 

In the following detailed portion of the present 
description, the invention will be explained in more 
15 detail with reference to the exemplary embodiments shown 
in the drawings, in which: 

Fig. 1 illustrates a preferred embodiment of a hand-held 
device according to the invention, 
20 Fig. 2 shows a block diagram of the hand-held device of 
the embodiment of Fig. 1, 

Fig.. 3 indicates the axes of movement and rotation along 
which the hand-held device is moved and rotated in order 
to create user input in accordance with the present 
25 invention. 

Figs . 3.1 to 3.7 illustrate the use of the present 
invention for zooming, scrolling and rotating images shown 
on the display. 

Figs. 3.8 to 3.10 illustrate the use of the present 
30 invention for user input to a graphical user interface to 
scroll a table in a window and to resize a window. 
Pigs. 3.11 to 3.13 illustrate the use of the present 
invention for scrolling a magnifying window over the 
display. 
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Figs. 3.14 and 3.15 illustrate the use of the present 
invention for moving a part of an image by cutting an 
pasting. 

Fig. 3.16 illustrates the use of the present invention 

5 with a text editing application. 

Fig. 3.17 illustrates the use of the present invention 
with an application for entering musical notes in a stave. 
Fig. 3.18 illustrates the use of the present invention 
with a labyrinth game 

10 Fig. 3.19 illustrates the use of the present invention for 
controlling a video game. 

Fig. 3.20 illustrates the use of the present invention 
with an application for controlling the soimd settings of 
a music player funtion, 
15 Fig. 4 outlines a software flow diagram for zooming, 
scrolling and rotating images shown on the display, and 
Figs. 5 and 6 show an alternative preferred embodiment of 
a hand-held device according to the invention. 

20 DETAILED DESCRIPTION 

This invention allows hand-held commvmication or computing 
devices with a relatively small display to receive user 
input by moving or rotating the device. In particular with 

25 devices having a display, the invention allows convenient 
navigation of a large stored virtual display or of objects 
in a graphical user interface. Such devices may include 
PDA devices, camcorders, digital photo cameras, digital 
binoculars (solid-state stereoscopic imaging system 

30 incorporated for viewing and digitally recording magnified 
stereo images), mobile hand-held terminals, advanced 
pagers, mobile telephones, and communicators. 

According to a preferred embodiment, the hand-held device 
35 is a hand portable phone, preferably a cellular /mobile 
phone . 
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Fig. 1 shows a mobile phone according to the invention, 
and it will be seen that the phone, which is generally- 
designated by 1, comprises a user interface having a 
5 keypad 2, a display 3, an on/off button 4 , a speaker 5 
(only the openings are shown) , and a microphone 6 (only 
the opening is shown) . The phone 1 according to the 
preferred embodiment is adapted for communication via a 
cellular network, such as the GSM 900/1800 MHz network . 

10 

The keypad 7 has a first group of keys 8 as alphanumeric 
keys, by means of which the user can enter a telephone 
number, write a text message (SMS), write a name 
(associated with the phone number) , etc. Each of the 
15 twelve alphanumeric keys 8 is provided with a figure "0-9" 
or a sign or respectively. In alpha mode each key 

is associated with a number of letters and special signs 
used in the text editing. 

20 The keypad 7 additionally comprises two softkeys 9, two 
call handling keys 12, and an arrow key 10. The function 
of the softkeys depends on the state of the phone and the 
navigation in the menu cam be performed by using the 
navigation-key. The present f\inction of the softkeys 9 is 

25 shown in separate fields in the display 3, just above keys 
9. The two call handling keys 12 are used for establishing 
a call or a conference call, terminating a call or 
rejecting an incoming call. This key layout is 
characteristic for e.g. the Nokia 6210™ phone. 

30 

The arrow key 10 is an up/down key which can be used for 
cursor movement and scrolling and is placed centrally on 
the front surface of the phone between the display 3 and 
the group of alphanumeric keys 7. A battery pack 14 is 
35 moiinted on the 'back of the phone and supplies electrical 
power for the electronic components of the mobile phone. 
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The phone has a flat display 3 that is typically made of a 
LCD with optional back lighting, such as a TFT matrix 
capable of displaying color images. 

5 

The phone is equipped with a digital camera 35 of which 
only the lens 36 is visible in Fig, 1. The camera is 
arranged at the rear of the phone above the battery pack 
14 • The camera direction is therefore siabstantially 

10 perpendicular to the rear surface of the phone 1 . Just 
under the camera an infrared port 3 8 for IrDA 
communication is provided (only window of the port is 
shown) - The camera may alternatively have a rotatable 
connection to the phone (not shown) , to allow adjustment 

15 of the camera direction relative to the housing of the 
phone • 

Fig. 2 schematically shows the most important parts of a 
preferred embodiment of the phone, in the form of a block 

20 diagram. A processor 18 controls the communication with 
the network via the transmitter /receiver circuit 19 and an 
internal antenna 20. A microphone 6 transforms the user's 
speech into cinalogue signals, the analogue signals foinned 
thereby are A/D converted in an A/D converter (not shown) 

25 before the speech is encoded in a digital signal 
processing unit 14 (DSP) . The encoded speech signal is 
transferred to the processor 18, which i.e. supports the 
GSM terminal software. The processor 18 also forms the 
interface to the peripheral units of the apparatus, 

30 including a RAM memory 17a and. a Flash ROM memory 17b, a 
SIM card 16, eind the keypad 2 (as well as data, power 
supply, etc.). The digital signal -processing xinit 14 
speech- decodes the signal, which is transferred from the 
processor 18 to the earpiece 5 via a D/A converter (not 

35 shown) . The processor communicates in two directions with 
the XrDA port 38 (infrared port) that allows data 
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cotnmvmication with other devices that are equipped with 
such a port, such a PC's, laptops, personal digital 
assistants (PDA) and other mobile phones. The phone may 
further be equipped with a short ramge RF transmitter 
receiver (not shown) , e.g. according to the Bluetooth 
standard, for data transmission with other devices as 
mentioned for IR data commimication. 



The processor 18 also forms the interface for the display 
10 controller 31. The image controller 31 receives the image 
files from the processor 18 and renders images for showing 
on the display 3 . 

A camera controller 37 is also connected to the processor 
15 18 and controls the digital camera 35. The camera 
controller 37 sets the resolution, the refresh rate, the 
focus, and zoom factor of the camera 35. The camera 
controller 37 sets the focus automatically through any of 
the well-known auto focus techniques available. The output 
20 signal of the camera 35 is connected to the processor 18. 



When the camera is used to detect motion of the hand-held 
device, the camera controller 37 automatically selects the 
appropriate resolution and refresh rate, so that the 

25 refresh rate is high enough to derive a smooth motion 
signal from the changes in the succeeding images. For 
motion detection it is not usually necessary to use the 
complete image captured by the digital camera. The 
software can pick out a particular section of the image 

30 for the motion detection so that the amount of data that 
has to be processed is reduced. These two measures (low 
resolution and using a section of the image) allow higher 
sampling rates and reduced power consumption because of 
lower data processing power demands. 



35 
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Some siirfaces in the camera view may not be particularly 
suited for detecting motion, e.g. because of a uniform 
surface, or because the distance to the objects in the 
camera view is too large to determine changes in distance 

5 accTirately. Such problems may be solved (in the embodiment 
with the adjustable camera direction) by directing the 
camera to another available object with sufficient 
texture, such as the user. The camera direction is thus 
reversed compared to the ""normal" direption. The motion 

10 signal derived from the camera signal is therefore 
automatically inversed when the camera is directed to the 
user. 

The camera 35 is a conventional digital camera and 
15 therefore not all the features of the camera 35 are shown. 
The image sensor of the digital camera can be any of the 
known configurations for solid-state image sensors, such 
as frame transfer, interline transfer CCDs, or diode 
arrays . 

20 

Standard CCD devices are sensitive to both visual light 
and near infrared light. Conventional digital cameras for 
capturing images with visual light are therefore provided 
with cin infrared filter for preventing influences of 

25 infrared light on the captured image. The mobile phone 1 
is also provided with an infrared filter (not shown) . The 
infrared filter can be moved out of the path of the light 
into the camera for capturing infrared images . The IrDA 
port 3 8 can be set to irradiate continuously to function 

30 as a light source for the camera 35 when it is in the 
infrared mode. Thus, the camera 35 can be used when there 
is little or no ambient light. The use of an IrDA port as 
a light sovirce for a digital camera and the details of a 
device to move the infrared filter in and out of the 

35 camera path as well adjustments to the auto focus system 
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are disclosed in US patent application witli serial nr. 
10/029,968, hereby incorporated by reference. 

The lens 3 6 is preferably a fixed focal length lens with 
5 movable lens group to allow auto focus, however, lens 3 6 
could be any type of lens providing for adjustment to 
focus on different parts of the image received, as will be 
understood to those skilled in the art. 

10 The characteristics of visible light and near infrared 
light with respect to focusing are slightly different. 
Therefore, the auto focus system has two settings; a first 
setting for capturing images with visual light and a 
second setting for infrared light. 

15 

Standard software for processing, storing and recalling 
pictures captured with visual light and captured with 
infrared light is installed on the phone 1. This software 
may as such be conventional and commercially available. 
20 The software is also able to control the refresh rate of. 
the images shown on the display. 

Optionally, the phone 1 may also comprise one or two tilt 
sensors 3 9 which determine the direction and magnitude of 

25 the displacement relative to vertical using the planetary 
gravitational field. Such sensors could be any of well 
known types such as those operating with an encoding disk 
on a freely rotatable shaft connected to a weight, or of 
the type that uses sphere provided with an asymmetrical 

30 weight that floats in a liquid. Alternatively the tilt 
sensor could by of the gyroscopic type. The signals of the 
camera the tilt sensors can be combined for generating the 
motion signal. 

35 The phone 1 in accordance with e preferred embodiment 
employs two operational modes associated with the use of 
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motion of the phone to generate user input, which 
throughout this document are referred to as navigation 
mode and fixed mode. When set to the navigation mode, 
motion of the phone 1 is used as input, and when 

5 displaying an image, the display view is automatically 
scrolled, zoomed and rotated to follow movements of the 
holding hand. The navigation mode is activated by pressing 
the left softkey 9 ^^Navigate" . Thus, the navigation mode 
is activated and the functionality of the left softkey 9 

10 changes to ^'Fixed" . When set back to the fixed mode by 
pressing the left softkey ^^Fixed" again, the display view 
becomes stationary and no longer follows the movements of 
the hand. 

n 

15 Fig. 3 indicates the relative axes of orientation along 
which the phone 1 is rotated or translated in order to 
navigate an image on the display 3 . Throughout this 
document, axis 40 will be referred to as the Y-axis, and 
motion of the phone in the direction of the Y-axis is in a 

20 preferred embodiment used to scroll images in the Y- 
direction. Similarly, axis 41 will be referred to as the 
X-axis and motion of the phone in the direction of the X- 
axis is in a preferred embodiment used to scroll images in 
the X-direction. Motion in the camera direction measured 

25 along the z-cocis 42 is in a preferred embodiment used to 
control the zoom factor of the images shown on the display 
3 . Rotation about the Z-axis is used to rotate images 
shown on the display 3. Though these are the preferred 
fxmctions assigned with movements along the X- ,Y- and Z- 

30 axes, any other functionality cam be assigned to movement 
in the direction of these sixes or to rotational movement 
about these axes. 

While the scrolling, zooming, and rotation of the display 
35 follows the movements of the device, the rate of 
scrolling/zooming/rotation as well as the amount of 
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scrolling/zooming/rotation need not follow the exact 
amount of change in position, and can be inverted. As will 
be discussed below, the control software of the present 
invention can smooth the movements of the phone to provide 
a convenient effect on the display. 

Figs . 3.1 to 3 . 13 show an overview of the operation of the 
phone 1 to scroll, zoom and rotate images. 



10 With reference to Fig. 3.1, a high resolution image of a 
holiday snapshot comprising a lake, a bridge and a 
moimtain peak stored in the RAM 17a is shown for viewing 
on the display 3 . The display 3 is too small to show the 
entire image with sufficient size to appreciate all the 

15 details in the image conveniently. The user presses the 
left softkey 9 ^^Navigate" to enter the navigation mode. 
The phone activates the navigation mode and changes the 
label above the left softkey 9 to *'Fixed" . In Fig. 3.2, 
-the navigation process is started when the operator's hand 

20 moves the phone 1 along the Z- axis 42 in the direction of 
arrow 42' for magnifying the view so that the display 3 
shows the central portion of the image with the bridge in 
an enlarged manner (Fig. 3.3). By moving the phone along 
the Y-axis 40 in the direction of arrow 40' the display 3 

25 scrolls the images upwards and the mountain peak above the 
bridge can be viewed (Fig. 3.5). 

In Fig. 3.6 the holiday snapshot is shown in a portrait 
orientation and Cctn only be properly viewed with the phone 

30 in a horizontal position. The user wishes to view the 
image with the phone in an upright position so that the 
image will be displayed in a landscape position with 
respect to the display 3 . The user presses the left 
softkey 9 ""Navigate" to activate the navigation mode and 

35 rotates the phone a half turn anticlockwise (Fig. 3.7). 
The movement of the phone 1 creates a series of changing 
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images captured by the camera 35 from which the software 
on the phone derives a motion signal. Upon detection the 
rotational movement about the Z-axis the software on the 
phone rotates the displayed image in the opposite 

5 direction, and when the user has completed a half 
anticlockwise turn the software has rotated the image a 
half turn clockwise, and the user can conveniently view 
the image with the phone in an upright position. The image 
is thus static with respect to the fixed coordinate system 

10 in which the phone 1 device is situated. 



The settings for the responses to motion of the phone 1 
can thus be set in way in which the user perceives the 
view as that of a static image over which a magnifying 
15 window (the display 3) is moved. 



Fig. 3.8 shows a display 3 of the phone in a mode in which 
a graphical user interface is used to command the device. 
A window 70 containing a scrollable table is displayed on 

20 the display 3. Scroll bars 71 and 72 are shown to the 
right and at the bottom of the table, respectively. A 
cursor 73 can be moved over the display by moving the 
phone in the direction of the X- and Y-axes. The left 
softkey 9 ^^Lef t-click" has the same functionality as the 

25 left mouse button as known from many windows based 
graphical user interfaces, namely to select a primary 
function associated with an object marked by the cursor 
73. By clicking, double clicking or holding the left 
softkey 9 down. The right softkey 9 '^Right- click" also has 

30 the same functionality as the right mouse button as known 
from many windows based graphical user interfaces, namely 
to select secondary functions associated with an object 
marked by the cursor 73. It is possible to assign the 
*^Left click" and "Right click" to other keys of the phone, 

35 but using a similar layout as for the keys of a computer 
mouse may facilitate user acceptance. 
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Example 1: The user wishes to scroll the table to the 
right to view column E. By holding the left softkey 9 
**Lef t-click" pressed and moving the phone to the right 
5 along the X-axis the table is scrolled by dragging it with 
the cursor to the right as shown in Fig . 3.9. 

Example 2: The user wishes to resize the window 70. After 
placing the cursor 73 on the upper bar of the window 7 0 
10 and while holding the left softkey 9 ""Lef t-click" movement 
of the phone 1 in the direction of the Z-axis resizes the 
window 70. Moving the phone towards the user enlarges the 
window as shown in Fig. 3.10 and moving away from the user 
reduces the size of the window (not shown) . 

15 

Example 3 : The user wishes to move a part of an image in 
an image editing program. The part of the image to be 
moved is marked with a sizable box 65. Holding the left 
softkey 9 "'Lef t-click" down and moving the mouse by 

20 rotating the phone about the X- and Y-axis resizes the box 
(Fig. 3.14) . After sizing the box 65 the left softkey 9 is 
released and the cursor 73 can be moved freely. The box 
can be dragged and dropped to the desired position by 
placing the cursor 71 in the box 65 and holding the left 

25 softkey 9 "Left-click" down whilst rotating the phone 
about the Y- and/ or X-axis vmtil the box has moved to the 
desired position. By releasing the left softkey 9 the 
boxed is dropped and the ciirsor 73 can move freely again. 
Thus, the part of the image in the box is cut from the 

30 original position and pasted to the new position. 

In the same or manner any object e.g. text in a text 
editor program, or numbers /text in a spreadsheet can be 
marked, resized, dragged and dropped ""click and drag" with 
35 the left softkey. 
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The most common ^^gestures" performed by moving the phone 
and pressing the softkeys are: 

point (to place the cvirsor over an object of the 

graphical user interface) , 



15 Figs . 3.11 to 3.13 show another method of manipulation 
magnified portion of an image. Fig. 3.11 shows the display 
3 with an image of a several advanced type mobile phones. 
A magnifying window 103 enlarges a portion of the image to 
allow the user a view with both a good overview and the 

20 possibility to view detail in a selected portion of the 
image. The magnifying window 103 can be moved over the 
display 3 by holding down the left softkey 9 "Navigate" 
whilst moving the phone in the direction of arrow 41' 
and/or arrow 40' (Fig. 3.12) to place the magnifying 

25 window at the desired position (Fig 3.13). The magnifying 
factor of the magnifying window can be changed by moving 
the phone 1 in the direction of the Z-axis (not shown) . 

Fig 3.16 shows another example of the use of the present 
30 intention in the form of an application for entering text. 
In the upper part of the display shows a string of 
characters already entered. A set of characters that can 
be entered, in this example the alphabet, is displayed 
below the upper part of the display. Other character sets 
35 could comprise a number set or special signs set, etc. 



10 



5 



left-click (to press and release the left softkey) 
to select the object which the cursor is placed, 
doiible- click (to press and release a softkey twice 
in rapid succession) to activate the object that the 
cursor is placed over, 

right -click (to press and release the right softkey) 
to call up a context-sensitive menu, and 
drag (to hold down the left softkey while moving the 
phone to move the cursor) to move or resize objects. 
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The fTonctionality of the left softkey 9 ^'Type" and the 
right softkey 9 ^'Options " is shown in the lower part of 
the display. One of the characters of the character set is 
marked by bold print. By rotating the phone about the Y- 
axis 40 the marking moves left or right. By rotating the 
phone about the X-axis 41 the marking moves up and down. 
The marked character is added to the string of characters 
by pressing the left softkey 9 ""Type" . 



10 By pressing the right softkey ""Options" a scrollable list 
of selectcLble menu items is displayed (not shown) 
comprising: ""Clear last character", ""Clear screen", 
""Number character set", ""Symbol character set", and 
""Exit" , one of the menu items being marked by inverse 

15 print. The list can be scrolled by rotating the phone 
about the Z-axis 41, and the marked menu item is selected 
by pressing the left softkey 9 ""Select" . 

Fig 3 . 17 shows another example of the use of the present 
20 invention in the form of an application for entering 
musical notes in a stave 69. A cursor 73 (shape changed to 
a cross for this application) is used to indicate the 
position where a note is to be entered. By rotating the 
phone about the Y-axis 40 the cursor can be moved left an 
25 right. By rotating the phone about the X-axis 41 the 
cursor can move up and down. 



The cursor is placed above the position in the stave at 
which a note is to be entered, . higher tones are placed 

30 higher up in the stave and lower tones are placed lower in 
the stave. A tone is entered by pressing the left softkey 
9 ""Type" . After typing a note the application prompts for 
entering the length of the note by displaying the text 
"^Length? 1=1 2=J$ 4:^M 8=^^/8" between the stave and the 

35 labels for the softkeys. The note length is entered by 
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pressing the alphanumeric key with the value associated 
with the desired note length. 

By pressing the right softkey '"Options" a scrollable list 
5 of selectable menu items is displayed (not shown) 
comprising: "'Clear last note", ""Clear stave", ""Enter 
special notes", and ""Exit", one of the menu items being 
marked by inverse print. The list can be scrolled by 
rotating the phone about the X-axis 41, and the marked 
10 menu item is selected by pressing the left softkey 9 
""Select". 

Fig. 3.18 shows another example of the use of the present 
invention in the fo2nn of an application simulating a 
15 labyrinth game. The player (user of the hand-held device) 
endeavors to guide a virtual ball 59 through a virtual 
labyrinth formed by virtual walls 57 on a virtual playing 
surface past a plurality of virtual openings 58 though 
which the virtual ball 59 may drop. 

20 

The application simulates the effect caused by gravity 
that tilting a real playing surface out of the horizontal 
plane has on a real ball, e.g. the virtual ball starts 
rolling to the lower side of the display 3 when the latter 
25 is tilted out of the horizontal plane. Also the effect of 
gravity on a real ball passing over a real opening is 
simulated by the application, e.g. the virtual ball drops 
through the virtual opening when it passes over a virtual 
opening . 

30 

At the start of the game, the display is held 
horizontally, or any other orientation that the user deems 
suitable as reference orientation to calibrate the virtual 
""horizontal" position of the display. The movement of the 
35 virtual ball over the virtual playing surface is 
controlled by moving the phone out of- and back into the 
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horizontal plane by rotating it rotating about the Y-axis 
40 and/or the X-axis. The rotational movements are 
detected from the changes between the succeeding images 
captured by the camera and translated into changes in 
speed and rolling direction of the virtual ball. 

Fig 3 . 19 shows another example of the use of the present 
invention in the form of an application for controlling 
another computer game, in this example a car racing 
simulation. The player (user of the hand-held device) 
endeavors to "^drive" a car around a racing circuit as fast 
as possible. The application allows the player to control 
the steering breaking and giving gas functions. The 
imaginary view on the racing circuit is the main content 
of the display, but is not shown on Fig. 3.19. The display 
further shows a steering wheel and rearview mirrors. A 
race is started by double clicking the left softkey 9 
^Action". The control settings for speed and directions 
are calibrated and set to zero for the present position 
and orientation of the phone 1. After an audible start 
signal the driver is supposed to attempt to follow the 
displayed racing circuit. The *'car" is steered by rotating 
the phone about the z-axis 42. Rotating the phone 
clockwise out of the calibrated position about the Z-axis 
makes the ^car" turn right. Rotating the phone further out 
of the calibrated orientation, make the ^^car" turn sharper 
and vice versa. Similarly, rotating the phone anti- 
clockwise about the Z-axis out of the calibrated position 
msikes the ''car" turn left. The speed of the ''car" is 
controlled by tilting the phone 1 about the X-axis 41. 
Rotating the phone 1 out of the calibrated position in one 
direction is used to give "gas". Rotating the phone 1 
further out of the calibrated orientation increases the 
amount of '^gas", and vice versa. The amount of breaking 
applied to the car is similarly controlled by rotating the 
phone 1 out of the calibrated position in the opposite 
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direction. The user may select which direction of rotation 
about the X-axis is used to give ""gas", whereby the 
breaking direction is always the opposite. Other computer 
games that can be controlled in a similar manner but using 

5 more axes of motion/ control include motor bike racing, and 
helicopter flying. For motorbike racing the factor balance 
can be included in relation to motion about one of the 
axes, to produce a very realistic experience, with e.g. 
steering bar rotation connected to rotation about the 

10 axis, giving gas and breaJcing connected to rotation about 
the X-axis and balance connected to translative motion 
along the X-axis. For enhancing the games the capacity of 
the mobile phone to generate soxind via the loudspeaker, in 
particular a hands- free loudspeaker is used to simulate 

15 e.g. motor, and/ or propeller sound. The vibrator function 
(not shown) of the mobile phone can be used to give 
feedback in connection with shocks and crashes. 

Fig 3.20 shows another example of the use of the present 

20 invention in the form of an application for controlling 
the soimd setting of a music player function of the phone. 
The application allows a user to control volume, bass and 
treble. The application shows a volume button, a bass 
button and a treble button on the display 3 . The button 

25 that is to be manipulated is be marked by a higher line 
thickness (as shown for the "Volume" button in Fig. 3.20). 
The marking can be moved to other buttons by rotating the 
phone about the Y-axis 40. The marked button is 
manipulated by holding the left softkey 9 *^Control" down 

30 while rotating the phone about the Z-axis 42. Clockwise 
rotation of the phone results in an increased setting of 
the parameter concerned, anticlockwise rotation of the 
phone results in a decreased setting of the parameter 
concerned. The application ends by pressing the right 

35 softkey 9 **Exit" . Alternatively, the display shows slide 
control knobs for each of the parameters to be controlled 
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(not shown) . The knob that is to be manipulated is be 
marked by a higher line thickness. The marking can be 
moved to other knobs by rotating the phone about the Y- 
axis 40. The marked knob is manipulated by holding the 

5 left softkey 9 ^^Control" down while moving the phone in 
the direction of the Z-axis 42. Moving the phone in the 
direction in which the display 3 is facing increases the 
parameter setting concerned, moving the phone in the 
opposite direction results in a decreased setting of the 

10 parameter concerned. 

Another example of the use of the present invention is in 
connection with another terminal such as a PC (not shown) . 
The motion signal of the phone is transmitted to the PC to 

15 control the movement of an object. The object could be a 
3-D object displayed on a screen connected to the PC, 
whereby orientation changes of the phone are used to 
change the orientation of the displayed object. The 
orientation of the object on the screen can be completely 

20 synchronized with the orientation of the phone. After an 
initial calibration of the relative positions, e.g. when 
the phone is upright, the displayed object is also 
upright. For e.g. presenting a product, the product can be 
shown as an object on a large screen. To change the 

25 orientation of the object the user changes the orientation 
of the phone by rotating it, and the PC rotates the 
displayed object in the same way in response to the signal 
that it receives from the phone. 

30 Another example of using the phone with another teinninal, 
i.e. a workstation or a PC is for moving through an 
imaginary 3-D space displayed on a screen by e.g. a CAD 
program. The movements of the phone in the real 3-D world 
are incorporated in the signal that is sent to the 

35 workstation and the PC or workstation translates the 
signal to movements of the viewing position in the **3-D 
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space" displayed on the screen. Clianges in the orientation 
of the phone are also incorporated in the signal that is 
sent to the workstation and translated to changes of the 
viewing direction in the space" displayed on the 

5 screen. Thus the user can ^^walk" through an imaginary room 
by walking aroxind in* the real world whilst holding or 
carrying the phone, amd change the viewing direction by 
pointing the phone in the desired viewing direction. To 
facilitate this manner of use, the invention could be 

10 incoirporated in a smaller device that is integrated in a 
helmet or mounted to another kind of headgear, so that 
viewing direction can be changed by the user turning 
his/her head in the desired direction of view. 

15 This manner of controlling the viewing position and the 
viewing direction can equally well be used for computer 
games and any other software applications that display a 
virtual 3--D space on a screen, i.e. to move through and 
observe a room, or any other place created in a virtual 

20 world. Another example of a use of this manner of 
controlling is game application that guides, i.e. 
instiructs a user to follow a virtual path and checks 
through the motion signal if the user really travels this 
path. Such games could be used by a group of users with 

25 interactive mobile phones in a suitable open space such as 
a sports grovmd. 

Fig. 4 outlines the software flow diagram for the 
manipulation of an image as described with reference to 

30 Figs. 3.1 to 3.7. The flow from start 80 to end 98 is 
perfoirmed several times a second in a standard polling 
process of the processor 18. At the initialization step at 
block 82, the current view settings are marked- The label 
above the left softkey 9 is set to ^^Navigation" in block 

35 84 . The status of the left softkey 9 is checked in block 
86- If the left softkey 9 is pressed, the system is set to 
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the navigation mode in block 88 smd the label above the 
left sof tkey is chainged to ""Fixed" . 



At block 90 the camera image and auto focus settings are 
5 acquired, stored and compared to the previous readings. If 
a change in image or in auto focus setting is detected at 
block 92, the program derives the movement of -the phone 1 
from the changes in block 94 and computes the new view 
settings i.e. zoom factor, rotation angle and portion of 
10 the image to be displayed. It also instructs the display 
controller 31 to refresh the display 3 to show the new 
view and it saves the present camera image as the basis 
for comparison in the next iteration of the process. 

15 In block 96 the status of the left softkey 9 is checked 
and if it is pressed the process ends at block 98 until 
the program is polled again. If the check for the left 
softkey 9 is negative, the -program goes to step 90 and the 
above process repeats itself. ^ 

20 

The program can be set with different response curves for 
computing the new view setting in response to changes in 
camera image and/ or auto focus setting at block 94. Fine 
or coarse modes of response can be set by the operator or 

25 can be changed dynamically during the time the system is 
in the navigation mode. With fine response, the display 
view changes at a relatively slow rate in response to the 
movements of the phone. With coarse response, the display 
view chcuages rapidly in response to the movements of the 

30 phone. The response can also be inverted to adapt to user 
preferences . 



The functionality associated with a given type of motion 
of the phone 1 can be set. Thus, the user can e.g. set the 
35 program such that the cursor 73 can be moved up and down 
by rotation about the X- axis and can be moved left and 
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right by rotation eQDout the Z-axis. Given types of motion 
of the phone 1 can be associated with functionality that 
does not relate to the display 3, such as sound settings 
(e.g. volume and balance) and display settings (e.g. 
5 brightness, color balance and contrast) . 

Figs. 5 and 6 show an alternative preferred embodiment of 
a hand-held device according to the invention in the form 
of a commtmicator 101. The communicator 101 is basically 

10 built up in the same way as the mobile phone 1 though with 
a larger display 103 and a larger keyboard 107. A camera 
135 {only lens 136 is shown) is mounted in the back/bottom 
of the communicator and has the same functionality as the 
camera in the phone 1. The internal hardware is also build 

15 up in the same way as the phone 1, but with increased 
processing power and a larger memory. Movements of the 
communicator 101 in the direction of or about the X-axis 
141, the Y-axis 140 and Z-axis 142 have the same 
functionality as in the phone 1. 

20 

Although the present invention has been described in 
detail for purpose of illustration, it is understood that 
such detail is solely for that purpose, and variations can 
25 be made therein by those skilled in the art without 
departing from the scope of the invention. 

Thus, while the preferred embodiments of the devices and 
methods have been described in reference to the 
30 environment in which they were developed, they are merely 
illustrative of the principles of the inventions. Other 
embodiments and configurations may be devised without 
departing from the scope of the appended claims. 
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CIAIMS ; 

1. A hand-held device comprising a processor, a digital 
camera for capturing motion video or still images, and 
5 means for transf oinning a signal from the camera into a 
motion signal indicative of the motion of the hand-held 
device. 

2- A hand-held device according to claim 1, further 
10 comprising a user interface in which motion of the hand- 
held device is - through the motion signal derived thereof 
- used as a user input. 

3 . A hand-held device according to claim 1 or 2 , further 
15 comprising a display suitable for displaying captured 

images . 

4 . A hand-held device according to claim 3 , in which 
motion of a given type of the hand-held device is used to 

20 manipulate images shown at least in part on the display, 
preferably by moving the images in a manner substantially 
corresponding to the movement of the hand-held device. 

5. A hand-held device according to claim 4, in which a 
25 given type of motion the hand-held device is used to move, 

and/or zoom, and/or expand/collapse and/or rotate images 
displayed on the display. 

6 . A hand-held device according to claim 5 , in which 
30 motion siabstantially parallel to the plane of the display 

of the hcind-held device is used to scroll an image 
displayed on the display, and/or motion substantially 
perpendicular to the plane of the display is used to zoom 
an image displayed on the display and/or rotational motion 
35 of the hand-held device is used to rotate an image 
displayed on the display. 
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7. A hand-held device according to any of claims 4 to 6, 
in which the images are images previously captured by the 
camera . 

5 

8. A hand-held device according to any of claims 4 to 7, 
in which movement of image is inverted with respect to 
motion of the hand-held device. 

10 9. A hand-held device according to any of claims 2 to 8, 
in which the user interface comprises a graphical user 
interface, and wherein motion of the hand-held device is 
used as an input to the graphical user interface. 

15 10. A hand-held device according to claim 9, in which 
motion of the hand-held device is used to manipulate an 
object displayed by the graphical user interface, 
preferably by moving the object in a manner substantially 
corresponding to the motion or to the inverted motion of 

20 the hand-held device, whereby the object displayed by the 
graphical user interface can be, an icon, a dialogue box, 
a window, a menu or a pointer. 

11. A hand-held device according to claim 9, in which 
25 motion of a given type of the hand-held device is used to 

move, and/ or zoom, and/or expand/collapse and/or rotate 
objects displayed by the graphical user interface. 

12. A hand-held device according to claim 11, in which 
30 motion substantially parallel to the plane of the display 

of the hand-held device is used to scroll an object 
displayed by the graphical user interface, and/or motion 
substantially perpendicular to the plane of the display is 
used to zoom an object displayed by the graphical user 
35 interface and/or rotational motion of the hand-held device 
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is used to rotate an object displayed by the graphical 
user interface . 

13. A hand-held device according to any of claims 2 to 12, 
5 in which the digital camera is detachable. 

14. A hand-held device according to any of claims 2 to 13, 
in which the digital camera is movable relative to the 
hand-held device. 

10 

15. A hand-held device according to any of claims 2 to 14, 
in which the means for transforming a signal from the 
camera into a motion signal derives the motion signal from 
changes between succeeding images, or parts of succeeding 

15 images captured by the camera. 

16. A hand-held device according to any of claims 2 to 15, 
in which the camera has an autofocus system, whereby the 
focusing setting of the autofocus system is used for 

20 detecting movement in the camera direction. 

17. A hand-held device according to any of claims 2 to 16, 
further comprising at least one key, wherein the 
functionality of a motion type is dependent on the state 

25 of the at least one key. 

18. A hand-held device according to any of claims 2 to 17, 
in which rotational motion of the hand- held device about 
an axis substantially perpendicular to the display results 

30 in an inverse rotational movement of the image or 
graphical user interface object relative to the display, 
preferably in a manner such that the image or object is 
static with respect to the fixed coordinate system in 
which the hand-held device is situated. 

35 
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19. A hand-held device according to any of claims 2 to 18, 
in which the motion signal is used to adjust device 
settings, the device settings preferably comprising sound 
settings and display settings. 



20. A hand-held device according to any of claims 9 to 19, 
further comprising a keypad with at least a first- and a 
second key and the graphical user interface comprises a 
cursor, whereby motion of the hand-held device is used to 
10 position the cursor over an object of the graphical user 
interface and primary fxmctions associated with the object 
concerned are activated by pressing the first key and 
secondary functions associated with the object of the 
concerned are activated by pressing the second key. 



21. A. hand-held device according to claim 20, in which the 
functionality of the first key is associated with 
selection and activation of objects of the graphical user 
interface, and in which the functionality of the second 

20 key is preferably associated with calling up a context- 
sensitive menu. 

22. A hand-held device according to claim 21, in which 
selection of the object concerned is performed by pressing 

25 and releasing the first key, and activation of the object 
concerned is preferably performed by pressing and 
releasing the first key twice in rapid succession. 

23. A hand-held device according to claim 21 or 22, in 
30 which moving or resizing of the object concerned is 

performed by holding down the first key while moving the 
hand-held device to move the ciirsor. 

24. A hand-held device according to any of claims 20 to 
35 23, in which the first key and the second key are softkeys 

whereby the current functionality of the softkeys is 
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shown in the display, preferably in dedicated fields of 
the display. 

25. A hand-held device according to claim 24, in which the 
5 first key is placed below the display on the left side of 

the latter, preferably proximate to lower edge of the 
display, and the second key is placed below the display on 
the right side of the latter, preferably proximate to 
lower edge of the display. 

10 

26. A hand-held device according to any of claims 1 to 25, 
fiirther comprising at least one gravity based tilt sensor, 
and whereby the signal from the at least one tilt sensor 
is used in combination with the signal from the camera for 

15 creating the motion signal. 

27. A hand-held device according to claim 26, wherein a 
tilt sensor is associated with the X-axis and/or a tilt 
sensor is associated with the Z-axis. 

20 

28- A hand-held device according to claim 27, wherein the 
signal form the at least one tilt sensor is used to 
determine the absolute orientation of the handheld device 
relative to the direction of the gravitational pull . 

25 

29- A hand-held device according to any of claims 1 to 28, 
further comprising means for sending the motion signal to 
another terminal via cable, infrared waves or radio 
frequency waves . 

30 

30- A system comprising a hand-held device according to 
claim 29 and a terminal capable of displaying imaginary 
three-dimensional objects on a two-dimensional screen, 
said terminal comprising means to change the orientation 

35 of the displayed object in response to signals received 
from the handheld device, whereby orientation changes of 
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the hand-held device are translated to corresponding 
orientation changes of the displayed object. 

31. A system according to claim 30, in which position 
5 changes of the handheld device are translated to position 
changes of the displayed object. 



32. A system comprising a hand-held device according to 
claim 29 and a terminal capable of displaying an imaginary 

10 three-dimensional space on a two-dimensional screen, said 
terminal comprising means to change the viewing position 
in the imaginary three-dimensional space in response to 
signals received from the handheld device, whereby 
positional changes of the hand-held device are trsoislated 

15 to corresponding changes in the viewing position. 

33. A system according to claim 30, in which orientation 
changes of the handheld device are translated into 
corresponding changes in the viewing direction in the 

20 imaginary three-dimensional space. 

34 . A method for creating user input for a hand-held 
device that has a processor, a user interface and a 
digital camera for capturing motion video or still images 

25 comprising the steps of: 

determining motion of the hand-held device from 
the camera signal; 

using the determined motion of the hand-held device 
as an input for the user interface. 

30 

35. Use of a digital camera for capturing motion video or 
still images of a hand-held device that has a processor to 
produce a motion signal indicative of motion of the hand- 
held device. 



35 
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36. A haxid-held device comprising a processor, means for 
sensing motion of the hand-held device, a display, a 
keypad with at least a first- and a second key, a 
graphical user interface with objects and a cursor, and 

5 means for transforming the sensed motion of the handheld 
device into a signal suitable for moving the cursor over 
the display, 

37. A hand-held device according to claim 36, in which 
10 motion of the hand-held device is used to position the 

cursor over an object of the graphical user interface and 
primary functions associated with the object conceamed are 
activated by pressing the first key and secondary 
functions associated with the object concerned are 
15 activated by pressing the second key. 

38. A hand-held device according to claim 37, in which the 
functionality of the first key is associated with 
selection and activation of objects of the graphical user 

20 interface, and in which the functionality of the second 
key is preferably associated with calling up a context- 
sensitive menu. 

39. A hand-held device according to claim 38, in which 
25 selection of the object concerned is performed by pressing 

and releasing the first key, and activation of the object 
concerned is preferably performed by pressing and 
releasing the first key twice in rapid succession. 

30 40. A hand-held device according to claim 38 or 39, in 
which moving or resizing of the object concerned is 
performed by holding down the first key while moving the 
hand-held device to move the cursor and the object 
concerned in unison therewith. 
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41. A hand-held device according to any of claims 36 to 
40, in which the first key and the second key are softkeys 
whereby the current fxinctionality of the softkeys is 
shown in the display, preferably in dedicated fields of 

5 the display. 

42. A hand-held device according to claim 41, in which the 
first key is placed below the display on the left side of 
the latter, preferably proximate to lower edge of the 

10 display, and the second key is placed below the display on 
the right side of the latter, preferably proximate to 
lower edge of the display. 

43. A hand-held device according to any of claims 36 to 
15 42, in which said means for transf oGoning motion of the 

handheld device into a signal suitable for moving the 
cursor over the display comprises a tilt sensor and/or an 
image capturing device and/ or an accelerometer . 

20 44. A hand-held device according to claim 43, in which 
said image capturing device is a motion video or still 
image digital camera. 

45- A hand-held device according to any of claims 36 to 
25 44, further comprising means to transfoarm a signal from 
the camera and/or tilt sensor and/or accelerometer into a 
position signal for the cxirsor. 

46. A hand-held device according to claim 45, in which 
30 said means for transforming a signal from the camera into 
a motion signal derives the motion signal from chcinges 
between succeeding images, or parts of succeeding images 
captured by the camera. 



35 4 7. A hand-held device according to any of claims 44 to 
46, in which the camera has an autofocus system, whereby 
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the focusing setting of the autofocus system is used for 
detecting movement in the camera direction. 

48. A hand-held device according to any of claims 36 to 
5 47, in which the graphical . user interface includes one or 
more of the following object types: icons, dialogue boxes, 
windows , menus , pointers . 



wo 2004/066615 



2/12 



T/EP2003/000607 



V 



20 



Transmitter / 

Receiver 

Circuit 

19 



Tiit sensor 39 



Digital signal 
processing 
unit 14 



Processor 



18 



▼ 



Camera 
controller 37 



I- Camera 35 



Microphone 6 



Speaker 



SIM card 



16 



Keyboard 



RAM 17a : 
Flash ROM 17b 



IrDA port 38 



Display 
controller 31 



Display 



Fig. 2 




Fixed Back 



Fig. 3.5 



wo 2004/066615 ^^T/EP2003/000607 

^ 6/12 



File1 




A 


B 


C 


D 


A_ 


1 












2 












3 












4 










▼ 


^1 






I- 





/7 

72 73 



Left-click 



70 



71 



Right-click 



Fig. 3.8 



File1 


B 


c 


D 


E 


F 
























1 

T 






















hi 1 , K 1 k 





/ 



72 73 



Left-click 



70 



71 



Right-click 



Fig. 3.9 



wo 2004/066615 ^^T/EP2003/000607 

7/12 



Left-click 



72 



73 









A 


B 


C 


D 


E ^ 


F 


▲ 


1 














2 














T 


3 














4 














5 














6 














^1 


► 









,70 



Right-click 



Fig. 3.10 




Left-click 



Right-dick 



-73 



Fig. 3.14 




Left-dick 




Right-dick 



3 
73 



Fig. 3.15 



HELLO 



ABCDEFGHi JKLiVIN 
OPQRSTUVWXYZ. 
Type Opttons 



® Fig. 3.16 



3 




9 



9 



Fig. 3.17 



wo 2004/066615 \^|^T/EP2003/000607 

8/12 





Fig. 3.13 



wo 2004/066615 



9/12 



;T/EP2003/000607 



Start 

in 



80 



Initialisation: mark cun^nt 

view settings 



Set left softkey label to 
"Navigation" and change 
to navigation mode 



82 




^ 84 



Change to navigation 
mode and change left 
softkey label to "Fixed" 



88 



Acquire camera image and 


^90 


auto focus setting and compare 




with previous image and auto 


^ 


focus setting 






No 



Change 
in camera image 
or zoom factor 
detected? 



Yes 



No 



Derive movement from change 
of camera image or change in 
auto focus setting and compute 
new view settings 




94 



Fig. 4 



wo 2004/066615 




:T/EP2003/000607 



10/12 




direction / 

< — > 



speed 



Action 



Exit 



oo o 

Volume Bass Tremble 



Control 



Exit 



9 




9 



Rg. 3.19 



9 



9 



Fig. 3.20 




Fig. 6 



INTEJ 




NAL SEARCH REPORT 



PCT/EP 03/00607 



A. CLASSIFICATION OF SUBJECT MATTER , 

IPC 7 H04N5/225 H04N7/14 H04M1/725 H04N5/232 H04M1/247 



According to Intemallonal Patent Ctassgicallon (IPC) or to both national classlRcatlon and IPC 



B. RELDS SEARCHED 



Minimum documentation seardied (classification system followed by dassmcation symbols) 

IPC 7 H04N H04M 



Documentation searched other than minimum documentation to the extent that such documents are Indudad in the fields searched 



Electronic data base consulted during the Internationa) search (name of data base and. where practical, search terms used) 

EPO-Internal , WPI Data, PAJ 



C. DOCUMENTS CONSIDERED TO BE RELEVANT 



Category * Citation of document, with Indication, where approprfate, of the relevant passages 



Relevant to claim No. 



wo 01 86920 A (LAPIDOT ZVI) 
15 November 2001 (2001-11-15) 



page 5, line 15 -page 11 



EP 0 884 905 A (NOKIA MOBILE PHONES LTD) 
16 December 1998 (1998-12-16) 
page 3, 11ne 39 -page 5, line 34 

EP 1 246 464 A (MITSUBISHI ELECTRIC CORP) 
2 October 2002 (2002-10-02) 
paragraph '0041! - paragraph '0066! 

./-- 



1-7. 

9-12,15, 
17,20, 
21,24, 
25, 

34-38, 

41-46 

16, 

26-29,47 
16,47 



26-28 



m 



Further documents are listed In the continuation of box O. 



Patent family memt>ers are listed In annex. 



' Special categories of died documents : 

'A' document defining the general state of the art which la not 
oonsldered to be of particular relevance 

'E' earner document but published on or after the International 
ffllng date 

•L' document which may throw doubts on priority clalmCB)or 
which ts cSed to establish the publlcailon date of another 
dtafion or other special reason (as apeclfled) 

*(y documenlrefentng to an oral disclosure, use, exhibition or 
other means 

*P' document published prior to the International filing date but 
later than the prbrtty date claimed 



"T" later document published after the International filing date 
or priority date and not In confflci with the appitoallon but 
dted to understand the principle or theory underly^Q the 
Invention 

*X* document of particular relevance; ttie claimed invention 
cannot be considered novel or cannot be considered to 
Involve an Inventive step when the document Is taken alone 

*Y* document of particular relevance; the claimed invention 

cannot be oonddered to Involve an Inventive step when the 
document Is combined with one or more other such docu- 
ments, such combination being obvious to a person akOled 
In the art. 

*&' document member of the same patent family 



Date of the actual conipletlon of the International search 



19 June 2003 



Dale of mainng of the Intematlonal search report 



26/06/2003 



Name and mailing address of the ISA 

European Patent Office. P.6. 5816 Patentlaan 2 
NL-2280HVRIlswi]k 
TeL (431-70) 340-2040, "Xtl 31 651 epo nl. 
Fax: (431-70) 340-3016 



Authorized officer 



Materne, A 



Foini KT/ISM210 (second sheet) guly 1982) 



lAL SEARCH REPORT 



"intemat^j^j^ 

PCT/EP 03/00607 



C^CentlnuaUon) DOCUMENTS CONSIDERED TO BE RELEVANT 



Category * Citation of documenl. with Indlcatlon.whsre appropriate, of the retevant passages 



Relevant to claim No. 



wo 01 97495 A (SLOTZNICK BENJAMIN) 
20 December 2001 (2001-12-20) 
page 8, line 20 -page 24 

EP 1 071 285 A (TEXAS INSTRUMENTS FRANCE 
;TEXAS INSTRUMENTS INC (US)) 
24 January 2001 (2001-01-24) 
column 5 -column 10, line 2 

WO 02 37179 A (KONINKL PHILIPS ELECTRONICS 
NV) 10 May 2002 (2002-05-10) 
page 3, line 14 -page 10 

WO 01 78055 A (FEINSTEIN DAVID Y) 
18 October 2001 (2001-10-18) 
dted In the application 
page 8, line 11 -page 22 



US 2002/160724 Al (ARAI FUJIO 
31 October 2002 (2002-10-31) 
page 2 -page 3 



ET AL) 



29 



1.34 



1,34 



1-48 



1-48 



Fbiin P0T/ISM210 (ooiltniallan at saeond dwrt) (July IBS) 



i 



INTEI 



InrSTniHtlon on pj 



AL SEARCH REPORT 

on patent family membera 



IntematMMViieatlon No 

PCT/EP 03/00607 



Patent document 
dtad In search report 


Publication 
date 


Patent family 
meml)er('8) 


Publloation 
data 


WO 0186920 


A 


15-11-2001 


AU 
WO 


5657601 A 
0186920 A2 


20-11-2001 
15-11-2001 



EP 0884905 


A 


16-12-1998 


FI 


972509 A 


14-12-1998 








CD 


nQQiionc AO 
Uoo4^Uo A2 










JP 


11075172 A 


16-03-1999 












VJl— Uo— tUUt 




A 

A 


U2-10-2U02 


ID 

Jr 




U4 iU— tUUt 








CN 


1378401 A 


06-11-2002 








EP 


1246464 A2 


02-10-2002 








NO 


20021509 A 


30-09-2002 








US 


2002142798 Al 


03-10-2002 


UO 0197495 


A 


20-12-2001 


AU 


7553801 A 


24-12-2001 








WO 


0197495 Al 


20-12-2001 








US 


2001055951 Al 


27-12-2001 




A 
M 




PP 

tr 




£H UX £.v\JX 


MO 0237179 


A 


10-05-2002 


wo 


0237179 A2 


10-05-2002 


UO 0178055 


A 


18-10-2001 


us 


6466198 Bl 


15-10-2002 








EP 


1290672 Al 


12-03-2003 








MO 


0178055 Al 


18-10-2001 








US 


2002190947 Al 


19-12-2002 


US 2002160724 


Al 


31-10-2002 


OP 


2001245267 A 


07-09-2001 








CN 


1363182 T 


07-08-2002 








GB 


2364194 A 


16-01-2002 








MO 


0163926 Al 


30-08-2001 



Rum PCT/ISA«10 (patent (amliy amox) (JuV 1892) 



This Page is Inserted by IFW Indexing and Scanning 
Operations and is not part of the Official Record 



Defective images within this document are accurate representations of the original 
documents submitted by the appUcant. 

Defects in the images include but are not limited to the items checked: 

□ BLACK BORDERS 

□ IMAGE CUT OFF AT TOP, BOTTOM OR SmES 

□ FADED TEXT OR DRAWING 

□ BLURRED OR ILLEGIBLE TEXT OR DRAWING 

□ SKEWED/SLANTED IMAGES 

□ COLOR OR BLACK AND WHITE PHOTOGRAPHS 

□ gray SCALE DOCUMENTS 

□ LINES OR MARKS ON ORIGINAL DOCUMENT 



□ OTHER: 



IMAGES ARE BEST AVAILABLE COPY. 
As rescanning these documents will not correct the image 
problems checked, please do not report these problems to 
the IFW Image Problem Mailbox. 



BEST AVAILABLE IMAGES 




REFERENCE(S) OR EXHIBIT(S) SUBMITTED ARE POOR QUALITY 



